Imaging method and system

ABSTRACT

An imaging method is provided. In accordance with this embodiment, an image of a scene is captured. Affective information is collected at capture. The affective information is associated with the image.

CROSS REFERENCE TO RELATED APPLICATIONS

[0001] Reference is made to commonly assigned U.S. patent applicationSer. No. 09/721,222, entitled “Method For Adding Personalized Metadatato a Collection of Digital Images” filed by Parulski et al. on Nov. 22,2000; Ser. No. 10/036,113, entitled “Method For Creating and UsingAffective Information in a Digital Imaging System” filed by Matraszek etal on Dec. 26, 2001; Ser. No. 10/036/123 entitled “Method for UsingAffective Information Recorded With Digital Images for Producing anAlbum Page” filed by Matraszek et al., on Dec. 26, 2001; Ser. No. ______Docket No. 85243 entitled “Camera System With Eye Monitoring” filed byMiller et. al; Ser. No. ______ Docket No. 84897 entitled “Method AndSystem For Creating And Using Affective Information In An Image CaptureDevice For Health Monitoring And Personal Security”; filed byFedorovskaya et al.; Ser. No. ______ Docket No. 85575 entitled “Methodand Computer Program Product For Determining an Area of Importance In AnImage Using Eye Monitoring Information” filed by Miller et al.; filedherewith, the disclosures of which are incorporated herein by reference.

FIELD OF THE INVENTION

[0002] The present invention relates to image capture systems and, moreparticularly, to image capture systems that also capture affectiveinformation.

BACKGROUND OF THE INVENTION

[0003] Increasingly, still and motion images are recorded in digitalform. Digital still and motion images can be captured using a digitalstill or digital video cameras. Digital still and motion images are alsoobtained by converting images that have been recorded in other ways intodigital form. For example it is well known to use analog to digitalconverters to convert analog electronic video signals into digitalimages. It is also known to use optical scanners to derive digitalimages from images recorded on photographic prints, films, plates andnegatives.

[0004] Digital still and motion images are easily viewed, stored,retrieved, and printed by a user using a home computer or other imageprocessing device. Such images can be uploaded to a website for viewing,as described in commonly assigned U.S. Pat. No. 5,666,215 filed byFredlund et al. on Aug. 3, 1995. Using a web browser, uploaded digitalimages can be viewed, selected for printing, electronically transmittedto other family members and/or friends or stored in on-line databasesand photo albums.

[0005] With the recent increase in the use of digital cameras forpicture taking and with the recent growth in the use of technology thatconverts conventional still images, analog video images, and film basedmotion pictures into digital form, the volume of digital images that areavailable is rapidly increasing. However, users often do not immediatelyprint or otherwise use the digital images, but instead opt to uploaddigital images to an electronic storage device or data storage mediumfor later use. Accordingly, personal computers, personal digitalassistants, computer networks, optical, magnetic and electronic storagemediums, so-called set top television devices and other electronic imagestorage devices are increasingly being used to store digital still andmotion images.

[0006] Therefore the task of classifying or cataloging digital still andmotion images on such storage devices in a way that they will be easilyaccessible by the user is becoming increasingly important. Some userscreate large personal databases to organize the digital still and motionimages on such storage devices. Many computer programs have beendeveloped to help users to do this. However, because of the time andeffort necessary to review and categorize images, these databases aretypically only rarely used and updated.

[0007] Thus, what is needed is a way to help organize and categorizeimages with reduced emphasis on the post capture analysis andcategorization of images.

[0008] Even when users make the investment of time and energy necessaryto organize images into databases, the databases are typically organizedaccording to various categories such as the date of capture, places,events, people. Other categories are also used. Often, such categoriesdo not inherently help the user to locate images that are of particularimportance or value. Instead the user must remember the image, and whenthe image was captured and/or how the user categorized it.

[0009] Thus, what is also needed is a more useful basis for organizingimages. It is known from various studies and observations that the mostmemorable categories of events and subsequently pictures are the onesthat are associated with user's feelings at the time of capture or theemotional reaction that the user experienced during the event or at thescene. Information that can be used to characterize the emotional stateof a user is known as affective information. Affective informationrepresents a user's psychological, physiological, and behavioralreactions to an event. Affective information can refer both to recordedraw physiological signals and their interpretations. Using affectiveinformation, digital still and video images can be classified based on auser's subjective importance, a degree of preference or the intensity ofand nature of specific emotions. Such classifications can help toquickly find, review and share those valuable images.

[0010] Various methods are known in the art for deriving affectiveinformation based upon a user's reaction to an image. One example of asystem that monitors physiological conditions to derive affectiveinformation is a wearable capture system that enables the classificationof images as important or unimportant based on biosignals from humanbody. This system was described in an article entitled “HumanisticIntelligence: “WearComp” as a new framework and application forintelligent signal processing” published in the Proceedings of theInstitute of Electrical and Electronics Engineers (IEEE), 86, pp.2123-2151, 1998 by Mann. In his paper, Mann described an example of howthe system could potentially operate in a situation when a wearer wasattacked by a robber wielding a shotgun, and demanding cash. In thiscase, the system detects physiological signals such as a sudden increaseof the wearer's heart rate with no corresponding increase in footsteprate. Then, the system makes an inference from the biosignals about highimportance of the visual information. This, in turn, triggers recordingof images from the wearer's camera and sending these images to friendsor relatives who would determine a degree of a danger.

[0011] Another example of such a system is described in a paperentitled, “StartleCam: A Cybernetic Wearable Camera” published in:Proceedings of the Second International Symposium on Wearable Computers,1998, by Healey et al. In the system proposed in this paper, a wearablevideo camera with a computer and a physiological sensor that monitorsskin conductivity are used. The system is based on detecting a startleresponse—a fast change in the skin conductance. Such a change in theskin conductance is often associated with reactions of sudden arousal,fear or stress. When the startle response is detected, a buffer ofdigital images, recently captured by the wearer's digital camera, issaved and can be optionally transmitted wirelessly to the remotecomputer. This selective storage of digital images creates a “memory”archive for the wearer which aims to mimic the wearer's own selectivememory response. In another mode, the camera can be set to automaticallyrecord images at a specified frequency, when very few responses havebeen detected from the wearer, indicating that their attention level hasdropped.

[0012] The systems proposed by Mann et al. make use of the physiologicalsignals to classify images as “important” (i.e., causing rapid change ina biological response) or “unimportant” (i.e., not causing rapid changein a biological response), and trigger the wearable camera to storeand/or transmit only the “important” images. However, their systems haveseveral shortcomings.

[0013] The described systems do not associate, do not store, and do nottransmit the physiological signals, or any other “importance” identifiertogether with the corresponding images. As a result, the “important”images can be easily lost among other images in a database, since thereis nothing in these “important” images to indicate that these images are“important”. This can happen, for example, when the digital image filesare used on a different system, when the images are transferred via arecordable contact disk or other media, when the images are uploaded toan on-line photo service provider, etc. The described systems also donot associate, do not store, and do not transmit the user's identifiertogether with the corresponding images. Therefore, when the system isused by more that one user, it is unable to distinguish which userreacts to the image as “important”.

[0014] Further, the described systems provide only binary classification“important-unimportant” and do not allow a finer differentiation of therelative degree of importance between the captured images. As a result,after a certain time of acquiring images in the user's database, thenumber of important images becomes too large to serve the purpose of theimportance attribute, unless the user will change the attribute forevery image in his or her database, which is a lengthy and tiresomeprocess.

[0015] Additionally, the described systems provide image classificationonly based on the general “importance” attribute. For example, they areunable to differentiate whether the important image evoked a positive(happy) or negative (unhappy) reaction in the user. Therefore, a widerange of human emotional reactions (e.g., joy, sadness, anger, fear,interest, etc.) is not considered in the system and cannot be applied tothe advantage of the user.

[0016] Consequently, a further need exists for an improved method forobtaining affective information and for using the affective informationto facilitate storage and retrieval of images.

SUMMARY OF THE INVENTION

[0017] In one aspect of the present invention, an imaging method isprovided. In accordance with this embodiment, an image of a scene iscaptured. Affective information is collected at capture. The affectiveinformation is associated with the scene image.

[0018] In another aspect of the invention another embodiment of animaging method is provided. In this embodiment, an image of a scene iscaptured. Affective signals are collected at capture. The relativedegree of importance of the captured image is determined based at leastin part upon the collected affective signals. The relative degree ofimportance is associated with the scene image.

[0019] In another aspect of the invention a photography method isprovided. In accordance with this method an image of a scene iscaptured. An image of at least a part of a photographer is obtained atcapture. Affective information is determined based at least in part oninterpretation of the image of the photographer. Affective informationis associated with the scene image.

[0020] In still another aspect of the invention, an imaging method isprovided. In accordance with this method, a stream of images is capturedand a corresponding stream of affective information is collected duringimage capture. The stream of affective information is associated withthe stream of images.

[0021] In yet another aspect of the invention, a method for determiningaffective information is provided. In accordance with this methodaffective signals are obtained containing facial characteristics andphysiological characteristics of a person. The facial characteristicsare analyzed and the physiological characteristics are analyzed. Theemotional state of the person is determined based upon analysis of thefacial and physiological characteristics of the person.

[0022] In still another aspect of the invention, an imaging system isprovided. The imaging system has an image capture system adapted tocapture an image selected by a user. A memory stores the image. A set ofsensors is adapted to capture affective signals from the user atcapture. A processor is adapted to associate the affective informationwith the captured image.

BRIEF DESCRIPTION OF THE DRAWINGS

[0023]FIG. 1a shows a handheld embodiment of an image capture system inaccordance with the present invention;

[0024]FIG. 1b shows a wearable embodiment of an image capture system inaccordance with the present invention;

[0025]FIG. 1c shows another wearable image capture system for creatingaffective information in association with a scene at the moment ofcapture;

[0026]FIGS. 2a and 2 b comprise a flow diagram showing one embodiment ofthe invention where providing affective information is provided based onanalysis of facial expressions;

[0027]FIGS. 3a and 3 b comprise a flow diagram showing an embodiment ofa the invention where affective information is provided based onanalysis of facial expressions;

[0028]FIGS. 4a and 4 b comprise a flow diagram showing an embodimentmethod where affective information is provided based on analysis offixation time;

[0029]FIGS. 5a and 5 b comprise a flow diagram showing a method whereaffective information is provided based on analysis of skin conductance;

[0030]FIGS. 6a and 6 b comprise a flow diagram showing an embodiment ofa method where affective information is provided based on combinedanalysis of facial characteristics and physiological characteristics;

[0031]FIGS. 7a and 7 b comprise a flow diagram showing anotherembodiment of a method for providing affective information based oncombined analysis of facial characteristics and physiologicalcharacteristics.

DETAILED DESCRIPTION OF THE INVENTION

[0032] The present invention provides a method for collecting affectiveinformation as a user views a scene and associating this information andits interpretation with a captured image of the specified scene.Interpretation of affective information can provide several gradationsof user's preference (e.g., the degree to which the user likes thescene). It also can provide a relative degree of importance of the sceneto the user. Additionally, interpretation of affective information canbe done in terms of the specific emotion (e.g., happiness, sadness,fear, anger, etc.) evoked by the scene.

[0033] A scene is defined as something seen by a viewer. It can be theplace where an action or event occurs, an assemblage of people and/orobjects seen by a viewer, a series of actions and events, a landscape orpart of a landscape, etc. Scenes recorded or displayed by an imagecapture device are referred to as images of scenes. Examples of imagecapture devices include digital still cameras, a handheld video cameras,a wearable video cameras, a conventional photographic cameras thatrecord images such as still or motion picture images on a film, ananalog video camera, etc. The user can observe scenes directly, througha camera's viewfinder, or on a camera preview screen serving as aviewfinder.

[0034] As used herein the terms image and images include but are notlimited to still images, motion images, multi-perspective images such astereo images or other depth images, and other forms of immersive stilland motion images.

[0035] People capture images of different scenes for a variety ofpurposes and applications. Capturing memorable events is one example ofan activity that ordinary people, professional photographers, orjournalists alike have in common. These events are meaningful oremotionally important to an individual or a group of individuals. Imagesof such events attract special attention, elicit memories, and evokeemotions, or, in general terms, they produce psychological reactions.Often these psychological reactions are accompanied by physiologicaland/or behavior changes.

[0036] Information that represents user's psychological, physiological,and behavioral reactions to a particular scene or an image of the scene,is referred to herein as affective information. Affective informationcan include raw physiological and behavioral signals (e.g., galvanicskin response, heart rate, facial expressions, etc.) as well as theirpsychological interpretation (e.g., preferred, not preferred, etc.), andassociation with an emotional category (e.g., fear, anger, happiness,etc.). The affective information is changed when a user's psychologicalreaction is changed. This can happen, for example, when a user suddenlysees a dangerous accident, an amazing action, or a beautiful landscape.

[0037] Affective tagging is defined as the process of determiningaffective information, and storing the affective information inassociation with images of a particular scene. When the affectiveinformation is stored in association with user identification data, itis referred to herein as “personal affective information”. The useridentification data can be any type of information that is uniquelyassociated with a user. The user identification data can be a personalidentification code such as a globally unique ID (GUID), user number,social security number, or the like. The user identifier can also be acomplete legal name, a nickname, a computer user name, or the like. Theuser identification data can alternatively include information such as afacial image or description, fingerprint image or description, retinascan, or the like. The user identification data can also be an internetaddress, cellular telephone number or other identification.

[0038] When the personal affective information is stored in associationwith the corresponding image, it is referred to as “personal affectivetag”. The affective information and user identifier are types of image“metadata”, which is a term used for any information relating to animage. Examples of other types of image metadata that can beincorporated in the personal affective information that is stored in theaffective tag include information derived from scene images andnon-image data such as image capture time, capture device, capturelocation, date of capture, image capture parameters, image editinghistory etc.

[0039] The personal affective information can be associated with adigital image by storing the personal affective information within theimage file, for example using a Tagged Image File Format IFD within anExif image file. Alternatively, the affective information can be storedin one or more application segments in a Joint Photographic Export Groupfile containing the first image (or alternatively the second image) inaccordance with the JPEG standard format ISO 10918-1 (ITU-T.81). Thisallows a single, industry standard image file to contain both a JPEGcompressed first image stored as a normal JPEG image, and the affectiveinformation to be stored in a proprietary form that is ignored by normalJPEG readers. In still another alternative, the personal affectiveinformation can be stored in a database that is separate from the image.This information can also be stored along with security and accesspermission information to prevent unauthorized access to theinformation.

[0040] Affective tagging can be done either manually or automatically,as a user views a particular scene or images of the scene using an imagecapture device. In the case of the manual affective tagging, user canenter affective information by using manual controls, which can includefor example, camera's control buttons, touch-screen display, or voicerecognition interface to provide his/her reaction to the scene. Forexample, in the case of a surprise, the user might “click” a camera'sbutton representing “surprise” reaction, or simply say a keyword such as“Wow!”.

[0041] In the case of automatic affective tagging, an image capturedevice can use one of the following affective signals or theircombinations to collect affective information, which can be subsequentlyinterpreted:

[0042] Eye movement characteristics (e.g., eye fixation duration, pupilsize, blink rate, gaze direction, eye ball acceleration, features andparameters extracted from the eye movement patterns, their complexity,etc.);

[0043] Biometric or physiological responses (e.g., galvanic skinresponse (GSR), hand temperature, heart rate, electromyogram (EMG),breathing patterns, electroencephalogram (EEG), brain-imaging signals,etc.);

[0044] Facial expressions (e.g., smile, frowns, etc.);

[0045] Vocal characteristics (e.g., loudness, rate, pitch, etc.);

[0046] Body gestures including facial movements (e.g., pinching bridgeof the nose, rubbing around ears, etc.).

[0047] In accordance with one embodiment of this invention describedbelow, affective information is determined automatically based on facialexpression, eye fixation duration, and galvanic skin response. Othercombinations can also be used.

[0048] Referring to FIGS. 1a-1 c, there are illustrated three exampleembodiments of image capture systems made in accordance with the presentinvention. The system depicted in FIG. 1a is a handheld image capturedevice 6 in possession of a particular user 2, who views a scene 4either directly, or through a viewfinder 24, or on a preview screen 22.It is understood that a digital still camera, handheld digital videocamera, wearable video camera, etc. may be considered as the imagecapture device 6. Examples of wearable embodiments of image capturedevice 6 are shown in FIG. 1b and FIG. 1c.

[0049] Image capture device 6 includes a capture module 8 to captureimages of the scene 4. The capture module 8 includes a taking lens (notshown), an image sensor (not shown) and an A/D converter (not shown).The capture module can also include a microphone (not shown), audioamplifier (not shown), and audio A/D converter (not shown). Capturemodule 8 provides digital still or motion image signals and associateddigital audio signals. Image capture device 6 also includes a centralprocessing unit (CPU) 14 and a digital storage device 12, that can storehigh-resolution image files such as digital still or digital motionimages provided by the capture module 8 as well as associated metadata.Digital storage device 12 can be a miniature magnetic hard drive, FlashEPROM memory, or other type of digital memory.

[0050] Image capture device 6 is shown adapted with a communicationmodule 18 such as a wireless modem or other communication interface thatexchanges data including digital still and video images using acommunication service provider, such as an Internet service provider 20.Communication module 18 can use a standard radio frequency wirelesscommunication systems, for example, the well-known Bluetooth system orthe IEEE Standard 802.15 system, a digital paging system, a conventionalcellular telephone system or other wireless systems. Alternativelycommunication module 18 can exchange information with other devicesusing infrared, laser, or other optical communication schemes. In stillanother alternative embodiment, image capture device 6 can have acommunication module 18 that is adapted to use data exchange hardwaresuch as a Uniform Serial Bus cable, IEEE Standard 1394 cable, otherelectrical data paths such as a wire or set of wires, a waveguide, or anoptical data path to permit information including digital images andaffective information to be exchanged between image capture device 6 andother devices.

[0051] In order to provide affective information, image capture device 6includes manual controls 13 and a set of sensors 15 that can detect auser's physiological signals. User 2 can enter affective information byusing controls 13, which can include for example, manual controlbuttons, touch-screen display, or a voice or gesture recognitioninterface.

[0052] Affective information can also be gathered by a set of sensors15. For example, in the embodiment shown in FIG. 1a, the set of sensors15 include galvanic skin response sensors 16 that are mounted on thesurface of the image capture device 6. In wearable embodiments any ofthe set of sensors 15 can be mounted elsewhere as shown in FIG. 1b,where galvanic skin response sensors 16 are located on the sidepiece 29of a conventional frame 28 used for supporting glasses. The set ofsensors 15 can also include a vascular sensor 17, usefully incorporatedon a portion of the sidepiece 29 proximate to the arteries in the templeof the head of the user thus facilitating measurement of temperatureand/or heart rate readings. The set of sensors 15 can also include avibration sensor 19 as is depicted in FIG. 1b proximate to the ears andcan be adapted to detect audible vibration proximate to the ear or byway of contact with the ear. Vibration sensor 19 can be adapted todetect both sounds emanating from the user and sounds that emanate fromother sources. Any of the set of sensors 15 can be located in otheruseful arrangements. Any one of the set of sensors 15 can beminiaturized so that their presence would not alter the appearance of awearable embodiment of image capture device 6. For example, as is shownin the embodiment of FIG. 1c sensors 16 for detecting galvanic skinresponse are a part of a wearable image capture device 6 mounted on abridge 26 of a conventional frame 28.

[0053] In other embodiments, the set of sensors 15 can comprise neuralsensors and other devices adapted to monitor electrical activity fromnerve cells to allow for interaction with the environment. Examples ofsuch sensors 15 include as the brain communicator and the MuscleCommunicator sold by Neural Signals, Inc., Atlanta Ga., U.S.A. Thesedevices monitor, respectively, electrical signals at a nerve cell andsignals radiated by certain nerves to detect the signals that are usedfor example to cause an average person to move an extremity. Thesesignals are transmitted to a computer, where software decodes thesignals into useful information. It will be appreciated that suchtechnology can be used to detect affective information as well as otherinformation useful in determining affective information. For example,neural activity along a nerve carrying sound information from an ear canbe monitored and used to determine audio information that reflects whatthe observer actually heard at an event.

[0054] Image capture device 6 also includes a user camera 10, which isused to record video images of eye movements, pupil size, and facialexpressions of the user 2. User camera 10 can incorporate for example aconventional charge couple device imager, a complimentary metal oxideimager or a charge injection device. Other imaging technologies can alsobe used. The images that are captured by user camera 10 can includevideo images for forming an image of the user or some feature of theuser's face. The images that are captured by user camera 10 can alsoinclude other forms of video images from which affective information canbe obtained. For example, images that represent eye position and pupilsize do not need to constitute full digital images of a user's eye.Instead other forms of imaging can be used that have lower resolution ora non-linear imaging pattern in order to reduce costs or to simplify theimaging structure

[0055] The video images captured by user camera 10 are stored on thedigital storage device 12 prior to processing by the CPU 14. User videocamera 10 can include, for example, an infrared sensitive camera. Inthis embodiment, a set of infrared light emitting diodes (infrared LEDs)direct infrared light toward the pupils of user. User video camera 10detects infrared signals radiated by the eyes of the user. The pupilsare then are tracked from the facial image of the user. One example of auseful user video camera 10 is the Blue Eyes camera system developed byInternational Business Machines, Armonk, N.Y., U.S.A. Another usefulexample of a user camera 10 is the Eyegaze System sold by LCTechnologies, Inc., Fairfax, Va., U.S.A. A version of the remotetracking eye-tracking camera ASL model 504 sold by Applied ScienceLaboratories, Boston, Mass., USA, can also be used. Other usefulembodiments of user camera 10 are shown and described in greater detailin commonly assigned U.S. patent application Ser. No. ______, (OurDocket # 85243) entitled “Camera System With Eye Monitoring” filedconcurrently herewith by Miller et al.

[0056] User video camera 10 can be attached to or located inside of thehandheld image capture device 6 as shown in FIG. 1a, on a head mountedframe 28 such as the wearable image capture device 6 as shown in FIG.1b, or on a remote frame of the wearable image capture device 6 as shownin FIG. 1c. In the case of FIG. 1c user video camera 10 is especiallysuitable for capturing a variety of facial features of the user,including pupil size, eye and brow movements. In the case depicted onFIG. 1b, it is best suited for capturing eye movements and other eyecharacteristics. User video camera 10 can also be separate from imagecapture device 6, and in this embodiment user video camera 10 cancomprise any image capture device that can capture an image of user ofimage capture device 6 and transfer this image to the image capturedevice. The transfer of images from a remote user video camera 10 can bedone wirelessly using any known wireless communication system.

[0057] Feature tracking can be performed using various algorithms, suchas for example, described in an article entitled “Facial FeatureTracking for Eye-Head Controlled Human Computer Interface”, published inProceedings of IEEE TENCON, 1999, pp. 72-75 by Ko et al). Thisalgorithm, capable of real-time facial feature tracking, composescomplete graph using candidate blocks it identifies from a processedfacial image, and then computes a measure of similarity for each pair ofblocks. The eyes are located as the blocks having the maximumsimilarity. Based on the eye position, the mouth, lip-corners andnostrils are located. The located features are tracked.

[0058] One example of a wearable image capture device 6 having a uservideo camera 10 that is adapted to record eye movements can be found,for example, in an article entitled “Oculomotor Behavior and PerceptualStrategies in Complex Tasks” published in Vision Research, Vol. 41,pp.3587-3596, [2001] by Pelz et al. The article describes a wearablelightweight eyetracker in the form of a head-gear/goggles, which includea module containing an infrared illuminator, a miniature video eyecamera, and a beam-splitter to align the camera to be coaxial with theilluminating beam. Retro-reflection provides the pupil illumination toproduce a bright-pupil image. An external mirror folds the optical pathtoward the front of the goggles, where a hot mirror directs the IRillumination toward the eye and reflects the eye image back to the eyecamera. A second miniature camera is mounted on the goggles to capture ascene image from the user's perspective.

[0059] In both FIG. 1b and FIG. 1c user video camera 10 is shown asconsisting consists of two pieces, which enable capture of eyecharacteristics of both eyes. It is however understood that user videocamera 10 may be represented by one piece that captures the eyecharacteristics of both or only one of the eyes of user 2.

[0060] Image capture device 6 is provided with appropriate softwarewhich is utilized by CPU 14 for creating and using personalizedaffective information. This software is typically stored on digitalstorage device 12, and can be uploaded or updated using communicationmodule 18. In addition, software programs to enable CPU 14 to performimage processing and analysis pertaining to non-affective information,which can be extracted from images of the scene provided by capturemodule 8, is also stored on the digital storage device 12. In addition,the digital storage device 12 can also store information with respect toa personal user profile, which could be a specific database thatincludes information summarizing characteristics of user's reactionssuch as, for example, quantitative information about typical reactionpatterns, as well as a software program to enable CPU 14 to access thisspecific database. This personal user profile can be queried by CPU 14when creating and using personalized affective information. The personaluser profile is updated by new information that is learned about thereactions of user 2.

[0061] It is understood that all parts and components of image capturedevice 6 discussed above may be implemented as integral parts of theimage capture device 6 or as physically separate devices connected withwires or wirelessly.

[0062] The following describes various embodiments of methods for imagecapture device 6 to determine affective information based on analysis offacial characteristics such as a degree of preference extracted fromfacial expression; or an emotional category and the degree ofdistinctiveness extracted from a facial expression. Other embodimentsshow methods for determining affective information based uponphysiological information such as a degree of interest extracted frompupil size and eye movements or a degree of excitement extracted fromgalvanic skin response. Further embodiments show methods for using acombination of facial analysis and physiological information to deriveaffective information.

[0063] Referring to FIGS. 2a and 2 b, there is shown a flow diagramillustrating an embodiment of a method of the present invention forproviding affective information based on the degree of preference of aparticular user for a subset of images of a particular scene. In thisembodiment, affective information is determined based on facialexpression of the particular user.

[0064] User 2 first activates the image capture device 6 (step 110). Ina preferred embodiment, the software application that implements themethod of the present invention is already installed in the imagecapture device 6 and it is launched automatically (step 112).Alternatively, the user can start the application manually, by usingappropriate control buttons (not shown) on the image capture device 6.

[0065] User 2 enters user identification data such as a user ID andpassword (step 114). In an alternative embodiment, the user video camera10 is used in conjunction with face recognition software toautomatically determine the identity of user 2, and to provide anappropriate user identifier, such as the user's name, personalidentification code or other identifier. In another alternativeembodiment user identification data can be obtained from data sourcesthat are external such as a radio frequency transponder to capturedevice 6 using, for example, communication module 18. In a furtheralternative embodiment, image capture device 6 is pre-programmed withuser identification data and step 114 is not required.

[0066] Image capture device 6 optionally provides a selection of signalsthat can be recorded in order to determine the user's emotional reactionas they view scenes (step 116). The user selects the desirable signal,i.e., facial expression in this case (step 118). In an alternativeembodiment, the image capture device 6 is preprogrammed to use one ormore affective signals, and steps 116 and 118 are not required.

[0067] User 2 then directs the imaging device to compose the scene to becaptured. The capture module 8 captures the first image of a scene (step120) and, simultaneously the user video camera 10 captures the firstfacial image of the user 2 (step 130).

[0068] Image capture device 6 temporarily stores the scene image (step122) and the facial image (step 132), and automatically analyses thefacial expression of user 2 (step 134). Facial expressions can beanalyzed using a publicly disclosed algorithm for facial expressionrecognition such as an algorithm described in an article entitled,“Facial Expression Recognition Using a Dynamic Model and Motion Energy”,published in Proceedings of the ICCV 95, Cambridge, Mass., 1995 by Essaet al. This algorithm is based on knowledge of the probabilitydistribution of the facial muscle activation associated with eachexpression and a detailed physical model of the skin and muscles. Thisphysics-based model is used to recognize facial expressions throughcomparison of estimated muscle activations from the video signal andtypical muscle activations obtained from a video database of emotionalexpressions.

[0069] Facial expressions can also be analyzed by means of otherpublicly available algorithms. One example of such an algorithm is foundin “Detection, Tracking, and Classification of Action Units in FacialExpression,” published in Robotics and Autonomous Systems, Vol. 31, pp.131-146, 2000 by J. J. Lien, et al. Another similar algorithm is foundin an article entitled “Measuring facial expressions by computer imageanalysis”, published in Psychophysiology, Vol. 36, pp. 253-263 byBartlett et al. [1999]. These algorithms are based on recognizingspecific facial actions—the basic muscle movements—which were describedin a paper entitled “Facial Action Coding System”, published inConsulting Psychologists Press, Inc., Palo Alto, Calif. by Ekman et al.[1978]. In the Facial Action Coding System (FACS), the basic facialactions can be combined to represent any facial expressions. Forexample, a spontaneous smile can be represented by two basic facialactions: 1) the corners of the mouth are lifted up by a muscle calledzygomaticus major; and 2) the eyes are crinkled by a muscle calledorbicularis oculi. Therefore, when uplifted mouth and crinkled eyes aredetected in the video signal, it means that a person is smiling. As aresult of the facial expression analysis, a user's face can berecognized as smiling when a smile on user's face is detected, or notsmiling when the smile is not detected.

[0070] Image capture device 6 determines the smile size (step 136). Ifthe smile is not detected, the smile size equals 0. If a smile has beendetected for a given image, a smile size for this image is determined asthe maximum distance between mouth corners within first three secondsafter the onset of the specified image divided by the distance betweenthe person's eyes. The distance between the eyes of user 2 is determinedusing the facial recognition algorithms mentioned above. The necessityof taking the ratio between the size of the mouth and some measurerelated to the head of user 2 (e.g. the distance between the eyes) stemsfrom the fact that the size of the mouth extracted from the facialimages depends on the distance of user 2 to user video camera 10,position of the head, etc. The distance between the eyes of user 2 isused to account for this dependency, however, other measures such as theheight or width of the face, the area of the face and other measures canalso be used.

[0071] Image capture device 6 determines the degree of preference (step138). If the smile was not detected, then the smile size andconsequently the degree of preference is equal to 0. If the smile wasindeed detected, the absolute degree of preference corresponds to thesmile size. The relative degree of preference is defined as the smilesize divided by an average smile size associated with a personal userprofile for user 2. The average smile size data in the personal userprofile can be constantly updated and stored on digital storage device12. The personal user profile with respect to average smile size can bethen updated using the smile size data (step 139).

[0072] The obtained degree of preference is compared to a thresholdvalue established by user 2. (step 140). If the obtained degree ofpreference is above the threshold value, then image capture device 6creates a personal affective tag for the corresponding image whichindicates a preference for this particular captured image (step 144). Inanother embodiment the threshold value for the degree of preferencecould also be established automatically from the personal user profile,for example, on the basis of the prior cumulative probabilities for theuser's degrees of preference distribution. In one embodiment such aprobability could be equal to 0.5, and thus, the threshold value for thedegree of preference would correspond to the value that occurs in atleast 50% of the cases. In yet another embodiment, the personalaffective tag can include a value selected from a range of preferencevalues, enabling the differentiation of the relative degree ofpreference between various captured images.

[0073] Image capture device 6 stores the corresponding image and thepersonal affective tag, which indicates the degree of preference, withinthe image file containing the scene image, as part of the image metadata(step 146). Alternatively, the personal affective tag, which indicatesthe degree of preference, can be stored in a separate file inassociation with the user identification data and the image identifier.When this is done, data is stored in the image metadata indicating thelocation of the file. In addition, the information about the date thatuser 2 views a certain image (i.e. immediately upon capture) can be alsorecorded as a separate entry into the personal affective tag.

[0074] In another embodiment the raw facial images are stored asaffective information either in a separate file on the image capturedevice 6 together with the image identifier and the user identifier, orin the personal affective tag as part of the image metadata, and theanalysis is done at a later time and optionally using a separate system.For example, the scene image and raw facial image can be communicatedusing the communication module 18 (see FIG. 1) and the Internet ServiceProvider 20 to a separate desktop computer (not shown) or computerserver (not shown), which can perform the analysis described earlier inrelation to steps 134-138.

[0075] The corresponding image, the personal affective tag and any otherimage metadata are sent using the communication module 18 to a personaldatabase of digital images (step 148). This personal database of imagescan be stored, for example, using separate desktop computer (not shown)or computer server (not shown).

[0076] In the embodiment shown, if the obtained degree of preference isbelow the threshold value the facial image of the user is deleted (step142). In another embodiment, if the obtained degree of preference isbelow the threshold value and if user 2 is still viewing the same sceneor a captured image of the scene, such as for example on preview screen22, image capture device 6 can optionally capture the next facial imageand repeat steps 132 through 140 to determine if user 2 has changed herfacial expression as user 2 views the same scene or the captured imageof the scene.

[0077] If the threshold value is set to 0, all scene images andcorresponding affective information (degree of preference or in anotherembodiment, raw facial image) recorded by the image capture device 6will be stored as described above.

[0078] If the user keeps the power turned on, (step 124) the process ofcapturing the next image of the scene (steps 120-124) are repeated andsimultaneously the steps of determining and storing a personal affectivetag for the captured image (steps 130-146) are repeated (step 126).

[0079] Image capture device 6 continues recording images of the scene 4using capture module 8 and facial images of user 2 using user videocamera 10, as long as user 2 keeps the image capture device 6 poweredon. If the power is turned off, image capture device 6 stops recordingthe images of the scene and the facial images and also ends the processof affective tagging (step 128).

[0080] The degree of preference can be used in a digital imaging systemto rank images in a systematic and continuous manner as favorite imagesfor a specified user as described in commonly assigned U.S. patentapplication Ser. No. 10/036,113, filed Dec. 26, 2001, entitled “Methodfor Creating and Using Affective Information in a Digital ImagingSystem” by Matraszek et al. and in commonly assigned U.S. patentapplication Ser. No. 10/036,123, filed Dec. 26, 2001, entitled “Methodfor Using Affective Information Recorded With Digital Images forProducing an Album Page” by Matraszek et al., the disclosures of whichare incorporated herein by reference.

[0081] In another embodiment, a binary degree of preference for imagesof a scene can be determined. When the smile is detected in step 136,the corresponding image is then classified as preferred with the binarydegree of preference equals 1. Alternatively, when the smile is notdetected, the image is classified as not preferred with the degree ofpreference equals 0.

[0082] The determined affective information in terms of the binarydegree of preference is then stored as a personal affective tag, whichincludes the user identification data as part of the image metadata. Itcan also be stored in a separate file on digital storage device 12together with the image identifier and the user identification data. Inaddition, affective information in terms of the actual image(s) of theuser's facial expression can also be stored in a separate file inassociation with the image identifier and the user identification.

[0083] In another embodiment, captured images are transferred by imagecapture device 6 to the Internet Service Provider 20 only when theaffective information exceeds a threshold, such as a threshold for therelative smile size. As a result, only images which exceed a preferencethreshold, are stored in the user's personal database of images. In thisembodiment, metadata is stored in the image files that indicate thatsuch files met the preference threshold.

[0084] Referring to FIGS. 3a and 3 b there is shown a flow diagramillustrating another embodiment of the present invention for providingaffective information that characterizes the emotional category of theuser's reaction, during image capture. In this embodiment, affectiveinformation is obtained based on the analysis of a user's facialexpressions.

[0085] Yet in another embodiment, an emotional category for images of ascene can be determined. The facial expression may be classified into abroader range of emotional categories, such as ‘happy’, ‘sad’,‘disgust’, ‘surprised’, etc. As a result of facial recognition, scenesthat evoke ‘happy’ facial expressions are assigned the ‘happy’ emotionalcategory; scenes that evoke ‘sad’ facial expressions are assigned the‘sad’ emotional category, etc. Images of the scene can be furtherclassified using a range of values for these categories, such asstrongly happy, somewhat happy, neutral and somewhat sad, and stronglysad, etc.

[0086] The determined affective information in terms of the emotionalcategory is then stored as personal affective tag, which includes theuser identifier as part of the image metadata. It can also be stored ina separate file on the digital storage device 12 together with the imageidentifier and the user identifier.

[0087] Facial expressions may be classified into a broader range ofemotional categories, such as ‘happiness’, ‘sadness’, ‘disgust’,‘surprise’, etc. A publicly disclosed algorithm that categorizes facialexpressions is described in an article entitled, “EMPATH: A NeuralNetwork that Categorizes Facial Expressions”, published in the Journalof Cognitive Neuroscience, 2002 by Dailey et al. The algorithmclassifies facial expressions into six basic emotional categories:‘happy’, ‘sad’, ‘afraid’, ‘angry’, ‘disgusted’, and ‘surprised’ based ondeveloping a feedforward neural network consisting of three neuronlayers performing three levels of processing: perceptual analysis,object representation, and categorization. In the model the first layermimics a set of neurons with the properties similar to those of complexcells in the visual cortex. The units in the second layer extractregularities from the data. The outputs of the third layer arecategorized into six basic emotions. As a result each facial expressionwill coded by six numbers, one for each emotion. The numbers,corresponding to different emotions are all positive and sum to 1, sothey can be interpreted as probabilities.

[0088] The following method determines an emotional category based on auser's facial expression, and further provides a range of values forthese categories, more specifically, the degree of “distinctiveness” ofan emotional category is suggested and shown in FIGS. 3a and 3 b. Thedegree of distinctiveness of an emotional category reflects a measure ofuniqueness or “purity” of a particular emotion as opposed to fuzzinessor ambiguity of the emotion. In common language such an emotion is oftenreferred to as “mixed feelings”.

[0089] Steps 150 through 172 of the embodiment of FIG. 3, generallycorrespond to steps 110 through 132 of the embodiment of FIG. 2.

[0090] In this embodiment image capture device 6 automatically analyzesthe facial expression of user 2 by applying the neural network methoddescribed by Dailey et al (step 174). As a result, a user's facialexpression is associated with six numbers, one for every basic emotion.

[0091] An emotional category (EC) is determined by choosing the categorywith the largest number (step 176). For example, if the numbers were0.5, 0.01, 0.2, 0.1, 0.19 and 0 for ‘happy’, ‘sad’, ‘afraid’, ‘angry’,‘disgusted’, and ‘surprised’, respectively, then the determinedemotional category is happy, because it has the largest respectivenumber 0.5. Consequently, scenes that evoke ‘happy’ facial expressionsare assigned the ‘happy’ emotional category; scenes that evoke ‘sad’facial expressions are assigned the ‘sad’ emotional category, etc.

[0092] Where several categories have the same number, one category canbe randomly selected to be the facial expression. Alternatively, whereseveral categories have the same number, other affective ornon-affective information can be used to help select a category.

[0093] Image capture device 6 determines the degree of distinctivenessof the emotional category (step 178). The degree of distinctiveness(DD_(EC)) is computed from the numbers for six emotions established inthe previous step 176, which are denoted for the convenience as N1, N2,N3, N4, N5, and N6. The following expression is used in the presentinvention for determining the degree of distinctiveness for theemotional category EC:

DD _(EC)={square root}{square root over ((N1² +N2² +N3² +N4² +N5²+N6²))}

[0094] DD_(EC) corresponds to the absolute degree of distinctiveness forthe emotional category EC. The relative degree of distinctiveness isdefined as the absolute degree of distinctiveness for the emotionalcategory EC divided by an average value for the DD_(EC) established in auser profile for the respective emotional category for the particularuser. The average DD_(EC) data in the user profile can be constantlyupdated and stored on digital storage device 12 as a part of a personaluser profile for user 2. The personal user profile is queried andupdated with respect to the average degree of distinctiveness of theemotional category DD_(EC) (step 179).

[0095] The obtained degree of distinctiveness is compared to a thresholdvalue established by user 2 or for user 2 (step 180). If the obtaineddegree of distinctiveness is above a threshold value, then, imagecapture device 6 creates a personal affective tag for the correspondingimage which indicates an emotional category with the degree of itsdistinctiveness for this particular captured image (step 184).

[0096] In another embodiment the threshold value for the degree ofdistinctiveness could also be established automatically from thepersonal user profile, for example, on the basis of the prior cumulativeprobabilities for the user's degrees of distinctiveness distributioncorresponding to a particular emotional category. In one embodiment suchprobability could be equal 0.5, and thus, the threshold value for thedegree of distinctiveness would correspond to the value that occurs inat least 50% of the cases. In yet another embodiment, the personalaffective tag can include a value selected from a range ofdistinctiveness values, enabling the differentiation of the relativedegree of distinctiveness between various captured images.

[0097] Image capture device 6 stores the corresponding image and thepersonal affective tag, which indicates the emotional category with thedegree of its distinctiveness, within the image file containing thescene image, as part of the image metadata (step 186). Alternatively,the personal affective tag, which indicates the emotional category withthe degree of distinctiveness, can be stored in a separate file inassociation with the user identification data and the image identifier.In addition, the information about the date that the user views acertain image (i.e. immediately upon capture) can be also recorded as aseparate entry into the personal affective tag.

[0098] In another embodiment the raw facial images are stored asaffective information either in a separate file on the image capturedevice 6 together with the image identifier and the user identifier, orin the personal affective tag as part of the image metadata, and theanalysis is done at a later time and optionally using a separate system.For example, the scene image and raw facial image can be communicatedusing the wireless modem 18 (see FIG. 1) and the Internet ServiceProvider 20 to a separate desktop computer (not shown) or computerserver (not shown), which can perform the analysis described earlier inrelation to steps 174-178.

[0099] The corresponding image, the personal affective tag and otherimage metadata are sent using the communication module 18 to InternetService Provider 20, to a personal database of digital images (step188). This personal database of images can be stored, for example, usingseparate desktop computer (not shown) or computer server (not shown).

[0100] In the embodiment shown, if the obtained degree ofdistinctiveness is below the threshold value the facial image of theuser is deleted. (step 182) If the obtained degree of distinctiveness isbelow the threshold value and if user 2 is still viewing the samecaptured image, such as for example on preview screen 22, image capturedevice 6 can optionally capture the next facial image and repeat steps172 through 180 to determine if user 2 has changed their facialexpression as user 2 views the same scene or a captured image on previewscreen 22.

[0101] If the threshold value is set to 0, all scene images andcorresponding affective information (emotional category with the degreeof distinctiveness or in another embodiment, raw facial image) recordedby the image capture device 6 will be permanently stored as affectiveinformation either in a separate file on the image capture device 6together with the image identification data and the user identifier, orin the personal affective tag as part of the image metadata.

[0102] Image capture device 6 continues recording images of the scene 4using capture module 8 and facial images of user 2 using user videocamera 10, as long as user 2 keeps the image capture device 6 poweredon. In step 168, if the power is turned off, the image capture device 6stops recording the images of the scene and the facial images and alsoends the process of affective tagging.

[0103] If user 2 keeps the power turned on, the process of capturing thenext image of the scene (steps 160-166) and simultaneously determiningand storing a personal affective tag for the captured image (steps170-186) are repeated.

[0104] The emotional category and its degree of distinctiveness can beused in a digital imaging system to rank images in a systematic andcontinuous manner as emotionally significant or favorite images for aspecified user as described in commonly assigned U.S. patent applicationSer. No. 10/036,113, entitled “Method for Creating and Using AffectiveInformation in a Digital Imaging System” by Matraszek et al. filed Dec.26, 2001 and in commonly assigned U.S. patent application Ser. No.10/036,123 entitled “Method for Using Affective Information RecordedWith Digital Images for Producing an Album Page” by Matraszek et al.,filed Dec. 26, 2001 the disclosures of which are incorporated herein byreference.

[0105] In another embodiment, only the emotional category for images ofa scene can be determined without the degree of distinctiveness. Whenthe emotional category is detected (step 174), the corresponding imageis then classified by the specified emotion. However, if two or moreemotional categories will have similar numbers produced by thecomputation in step 176, a neutral category can be assigned.

[0106] The determined affective information in terms of the emotionalcategory is then stored as personal affective tag, which includes theuser identification data as part of the image metadata. It can also bestored in a separate file on the digital storage device 12 together withthe image identifier and the user identification data. In addition,affective information in terms of the actual image(s) of the user'sfacial expression can also be stored in a separate file in associationwith the image identifier and the user identification data.

[0107] In previously discussed embodiments affective information wasextracted from facial characteristics of user 2. FIGS. 4a and 4 b show aflow diagram illustrating another embodiment of the present inventionwhere affective information is provided in terms of a degree of interestbased upon a physiological characteristic namely eye gaze information.With this embodiment, a degree of interest is determined based on eyegaze fixation time, which is the time that eyes of user 2 are fixated ata particular location of the scene, before fixating at a differentlocation.

[0108] The data described in a paper entitled “Looking at Pictures:Affective, Facial, Visceral, and Behavioral Reactions”, published byPsychophysiology, Vol. 30, pp. 261-273, by Lang et al., 1993, indicatesthat on average, viewing time linearly correlates with the degree of theinterest or attention an image elicit in an observer. Thus, such arelationship allows interpreting the fixation time as the user's degreeof interest toward an area of a scene. The quoted publication by Lang etal. compares a viewing time with the degree of the interest for thirdparty images of scenes only. In the present invention, fixation timeinformation is assessed directly for scenes as well as first partyimages of the scenes and stored as a personal affective tag as part ofthe image metadata or in a separate file in association with the useridentifier and the image identifier.

[0109] In the embodiment of FIGS. 4a and 4 b, method steps 210-228generally correspond to method steps 110-128 in FIG. 2 with only onedifference: in step 218, the user selects the “fixation time” signal.Alternatively, the image capture device 6 can be preprogrammed tocapture the “fixation time” information.

[0110] In this embodiment, user video camera 10 in image capture device6 captures a sample of eye images during a time window, such as a timewindow of 30 seconds, when the user views the scene (step 230) duringone of image composition, capture, and immediate post capture review. Insome embodiments, the time window can be modified by user 2.

[0111] Coordinates of the eye gaze direction of user 2 are stored with asampling rate, such as a sampling rate of 60 Hz (step 232). In someembodiments, the sampling rate can be modified by user 2. The samplingrate can also be modified based upon other factors such as the rate ofcharge from the eye gaze as the time rate of change of scene contents orthe amount of memory available for storing affective data.

[0112] The raw gaze coordinates are grouped into eye fixations (step234). An eye fixation is usually defined as period of at least 50 msecduring which the gaze coordinates do not change by more than 1-degree ofvisual angle. For each fixation, a start time, end time and gazecoordinates are determined. Additionally, an average pupil diameter canbe determined for each fixation. The duration of eye fixations ismeasured based on their start and end times (step 236).

[0113] Image capture device 6 determines the degree of interest for eacheye fixation (step 238). The absolute degree of interest is defined asthe corresponding fixation time. The relative degree of interest isdefined as the fixation time divided by the average fixation time forthe particular user. The average fixation time can be constantly updatedand stored on digital storage device 12 as a part of a personal userprofile for user 2. The personal user profile is queried and updatedwith respect to the average fixation time for user 2 (step 239).

[0114] The obtained degree of interest is compared to a threshold valueestablished for the user (step 240). If the obtained degree of interestis above the threshold value, then the image capture device 6 creates apersonal affective tag indicating the degree of interest (step 244).Alternatively, the threshold value for the degree of interest could alsobe established automatically from the personal user profile, forexample, on the basis of the prior cumulative probability for the user'sdegree of interest distribution of user 2. One such probability could beequal 0.5, and thus, the threshold value for the degree of interestwould correspond to the value that occurs in at least 50% of the cases.

[0115] In one embodiment, image capture device 6 stores thecorresponding image and the degree of interest in the personal affectivetag as part of the image metadata (step 246). Alternatively, the degreeof interest can be stored in a separate file in association with theuser identification data and the image identifier. Where this is done,data is stored in the image metadata indicating the location of the filehaving the personal affective information. In addition, the informationabout the date the user views a certain image can be also recorded as aseparate entry into personal affective tag.

[0116] In another embodiment the scene images and the raw eye images arestored. The raw eye images can be analyzed later, either by the CPU 14or by a processor in a separate device (not shown), which receives thestored images.

[0117] The corresponding image, the personal affective tag and otherimage metadata are sent to a personal database of digital images, asdescribed earlier in relation to step 148 (step 248).

[0118] If the obtained degree of interest is below the threshold valuethe corresponding eye images are deleted (step 242). If the obtaineddegree of interest is below the threshold value and user 2 is stillviewing the same scene or a captured image of the same scene such as forexample on preview screen 22, image capture device 6 can optionallycapture another segment of eye images and repeat steps 232 through 240as user 2 views the same scene or the captured image of the same scene.

[0119] If the threshold value is set to 0, all scene images andcorresponding affective information (degree of interest or, in anotherembodiment, raw eye images) recorded by the image capture device 6 willbe stored as affective information either in a separate file on theimage capture device 6 together with the image identifier and the useridentifier, or in the personal affective tag as part of the imagemetadata.

[0120] The image capture device 6 continues recording images f the scene4 using capture module 8 and facial images of user 2 using user videocamera 10, as long as user 2 keeps the image capture device 6 poweredon. In step 168, if the power is turned off, the image capture device 6stops recording the images of the scene and the facial images and alsoends the process of affective tagging.

[0121] If the user keeps the power turned on, the process of capturingthe next image of the scene (steps 220-226) and simultaneouslydetermining and storing a personal affective tag for the captured image(steps 230-246) are repeated.

[0122] As mentioned earlier, the degree of interest can be used in adigital imaging system to rank images in a systematic and continuousmanner as favorite or high value images for a specified user asdescribed in commonly assigned U.S. patent application Ser. No.10/036,113, filed Dec. 26, 2001, entitled “Method for Creating and UsingAffective Information in a Digital Imaging System” by Matraszek et al.U.S. patent application Ser. No. 10/036,123, filed Dec. 26, 2001,entitled “Method for Using Affective Information Recorded With DigitalImages for Producing an Album Page” by Matraszek et al.

[0123] In alternative embodiments, user camera 10 and central processingunit 14 can be used to obtain additional information from images ofuser's eye(s). Examples of such information include but are not limitedto eye ball acceleration, tear formation, eye temperature, irispatterns, blood vessel patterns and blood vessel size. This informationcan be used to determine user's identity, emotional state and/or healthcondition. This information can be stored as part of an affective tag.

[0124] Another source of affective information originates fromphysiological signals generated by user 2. FIG. 5 illustrates anembodiment of the present invention where affective information isdetermined from a physiological signal. In this embodiment, thephysiological signal is skin conductance and the affective informationderived from the skin conductance is expressed in terms of a degree ofexcitement.

[0125] Skin conductance change is a measure of galvanic skin response.Skin conductance reflects a change in a magnitude of the electricalconductance of the skin that is measured as a response to a certainevent—viewing the scene or images of the scene. As described in thepaper entitled “Looking at Pictures: Affective, Facial, Visceral, andBehavioral Reactions”, published in Psychophysiology, Vol. 30, pp.261-273, 1993, by Lang et al., skin conductance changes depends on thearousal the image elicits in the viewer: the higher the conductance, thelower the arousal or excitement, and vice versa: the lower theconductance, the higher the arousal. The measure of the amplitude of theskin conductance response is also used to determine interest orattention.

[0126] In this embodiment, method steps 310-328 generally correspond tosteps 110 through 128 in FIG. 2 with only one difference: in step 318,user 2 can manually instruct image capture device 6 to capture galvanicskin response information as at least a part of the affectiveinformation. Alternatively image capture device 6 can be preprogrammedto capture galvanic skin response information. The image capture device6 measures the galvanic skin response signal during a time window, forexample a time window of 5 seconds, using the galvanic skin responsesensor 16 (step 330). In some embodiments, the time windows can bemodified by user 2. One example of galvanic skin response sensor 16 canbe, for example, SC-Flex/Pro sensor from ProComp detector system sold byThought Technology, Ltd. W. Chazy, N.Y. USA.

[0127] The galvanic skin response skin conductance signals are storedusing a sampling rate, for example a sampling rate of 60 Hz (step 332).In some embodiments, the sampling rate can be modified by user 2. Thesampling rate can also be modified based upon other factors such as therate of change of scene contents, the time rate of change of galvanicskin response, or the amount of memory available for storing affectivedata. The galvanic skin response conductance signals are filtered toreduce the noise in the data (step 334). The amplitude of the galvanicskin response skin conductance signals is then determined (step 336).

[0128] Image capture device 6 determines the degree of excitement basedupon galvanic skin response signals (step 338). The absolute degree ofexcitement for the scene is equivalent to the amplitude of the filteredgalvanic skin response skin conductance signal. The relative degree ofexcitement is defined as the amplitude of the galvanic skin responsesignal divided by the average galvanic skin response signal for theparticular user. The average skin conductance can be constantly updatedand stored on digital storage device 12 as a part of the user'spsychophysiological profile. To compute the relative degree ofexcitement, the average skin conductance response information isretrieved from a personal user profile. The personal user profile isupdated regarding the skin conductance response information (step 339).

[0129] The obtained degree of excitement is compared to a thresholdvalue established for the user (step 340). If the obtained degree ofexcitement is above the threshold value, the image capture device 6creates a personal affective tag indicating the degree of excitement(step 344). In another embodiment, the threshold value for the degree ofexcitement can be established automatically from the personal userprofile, for example, on the basis of the prior probabilities for theuser's degrees of excitement. One such probability could be equal 0.5,and thus, the threshold value for the degree of excitement wouldcorrespond to the value that occurs in at least 50% of the cases.Alternatively, the personal affective tag can include a value selectedfrom a range of excitement values enabling the differentiate of therelative degree of excitement between various captured images.

[0130] Image capture device 6 stores the corresponding scene image andthe degree of excitement in the personal affective tag as part of theimage metadata (steps 344 and 346). Alternatively, the degree ofexcitement can be stored in a separate file in association with the useridentification data and the image identifier. In addition, theinformation about the date the user views a certain image also can berecorded as a separate entry into personal affective tag.

[0131] In another embodiment the raw galvanic skin response signal isstored as affective information either in a separate file on the imagecapture device 6 together with the image identifier and the useridentification data, or in the personal affective tag as part of theimage metadata.

[0132] The corresponding image, the personal affective tag and otherimage metadata are sent to a personal database of digital images, as wasdescribed earlier in relation to step 148 (step 348).

[0133] In step 342, if the obtained degree of excitement is below thethreshold value the corresponding skin conductance signals are deleted.If the obtained degree of excitement is below the threshold value anduser 2 is still viewing the same scene or a captured image of the scenefor example on a preview screen 22, image capture device 6 canoptionally measure the next segment of skin conductance signals for 5seconds and repeat steps 332 through 340 as user 2 views the same sceneor a captured image of the same scene.

[0134] If the threshold value is set to 0, all scene images andcorresponding affective information recorded by the image capture device6 will be stored. The personal user profile can then be updated (step339).

[0135] As mentioned earlier, the degree of excitement can be used in adigital imaging system to rank images in a systematic and continuousmanner as favorite, important or exciting images for a specified user asdescribed in commonly assigned U.S. patent application Ser. No.10/036,113, filed Dec. 26, 2001, entitled “Method for Creating and UsingAffective Information in a Digital Imaging System” by Matraszek et al.;Ser. No. 10/036,123, filed Dec. 26, 2001, entitled “Method for UsingAffective Information Recorded With Digital Images for Producing anAlbum Page” by Matraszek et al.

[0136] It is understood that each user 2 might have differentphysiological and facial responses to an image. Some users might exhibitstrong physiological responses while exhibiting only modest facialresponses. Other users might exhibit modest physiological responseswhile exhibiting strong facial responses. Still other users mightexhibit modest physiological and facial responses. Accordingly, bycombining different types of affective information, a more robustrepresentation of the emotional response of user 2 to the scene can beobtained. The following embodiments show methods for interpretingaffective information using physiological and facial responseinformation in combination to help facilitate interpretation ofaffective information.

[0137] Referring to FIGS. 6a and 6 b, there is shown a flow diagramillustrating another embodiment of the present invention for providingaffective information based on the combination of the three affectivesignals described in relation to FIGS. 2-4, namely, the degree ofpreference, the degree of interest, and the degree of excitement, whichare further combined to obtain an integral measure of positiveimportance.

[0138] In this embodiment, method steps 410-428 correspond to methodsteps 110-128 in FIG. 2 with only one difference: in step 418, the userselects the use of “combined” signals or alternatively, the imagecapture device 6 is preprogrammed to use the “combined” signals.

[0139] The image capture device 6 determines the degree of preference(DP) based on facial expression as was described earlier in relation tosteps 130 through 139 in FIG. 2. (steps 430 and 432) The image capturedevice 6 also determines the degree of interest (DI) based on fixationtime the same way as in steps 230 through 239 in FIG. 3. (steps 434 and436) The image capture device 6 further determines the degree ofexcitement (DE) based on skin conductance the same way as in steps 330through 339 in FIG. 5 (steps 438 and 440).

[0140] The image capture device 6 determines the degree of positiveimportance (or “favoriteness”) based on a sum of the three measures:

Positive Importance=DP+DI+DE

[0141] In another embodiment, the degree of positive importance isdetermined based on a weighted sum of these three measures,

Positive Importance=w _(DP) DP+w _(DI) DI+w _(DE) DE

[0142] where the weights w_(DP), w_(DI), and w_(DE) are determined basedon the standard deviation within each of the normalized (divided by themaximum value) signals previously obtained for the particular user. Inthis case, the higher the standard deviation within the signal, thehigher the weight of the contribution for the signal into the measure ofpositive importance. Consequently, the lower the standard deviation of agiven signal, the lower the weight of the contribution for thecorresponding signal into the measure of positive importance. The reasonfor this dependency stems from the assumption that a standard deviationof a particular measure for a particular user reflects an individualdegree of differentiation between different scenes. This implies thatthe signal with the highest standard deviation has more differentiationpower, and therefore is more important to consider while determining anintegral measure of positive importance for a particular user.

[0143] For example, if different scenes evoke a large variations offacial expression and a low variation of skin conductance responses fora user A, than the weight given to the measure of degree of preference(DP) based on facial expression w_(DP) would be higher than the weightgiven to the measure of the degree of excitement (DE) based on skinconductance w_(DE). On the other hand, if different scenes evoke smallervariations of facial expression and a large variation of skinconductance responses for a user B, than the relationships between theweights is reversed. Data about the maximum values and the standarddeviation of the corresponding signals can be obtained from the personaluser profile in step 443. The personal user profile is then updatedregarding this information.

[0144] The obtained degree of positive importance is compared to athreshold value (step 444). This threshold can be predetermined. Thethreshold can also be established by user 2 or established for user 2.If the obtained degree of positive importance is above the thresholdvalue, then the image capture device 6 creates a personal affective tagindicating the degree of positive importance (step 448). In anotherembodiment the threshold value could be determined automatically fromthe personal user profile, for example, on the basis of the priorcumulative probabilities of the degree of positive importancedistribution. One such threshold value then can be chosen ascorresponding to the degree of positive importance with the priorprobability of 0.5.

[0145] Image capture device 6 stores the corresponding scene image andthe degree of positive importance in the personal affective tag as partof the image metadata (step 450). Alternatively, the degree of positiveimportance can be stored in a separate file in association with the useridentification data and the image identifier. In addition, theinformation about the date the user views a certain image can be alsorecorded as a separate entry into personal affective tag.

[0146] The corresponding image, the personal affective tag and otherimage metadata are sent to a personal database of digital images, asdescribed earlier in reference to step 148 of FIG. 2 (step 452).

[0147] If the obtained degree of positive importance is below thethreshold value the corresponding segment of facial images, eye imagesand skin conductance signals are deleted (step 446).

[0148] If the obtained degree of positive importance is below thethreshold value and user 2 is still viewing the same scene or a capturedimage of the scene for example on preview screen 22, image capturedevice 6 can optionally measure the next segment of facial images, eyeimages and skin conductance signals and repeat steps 432 through 444 asuser 2 is views the same scene or the captured image of the same scene.

[0149] If the threshold value is set to 0, all scene images andcorresponding affective information (degree of positive importance or,in another embodiment, raw facial images, eye images and galvanic skinresponse signals) recorded by the image capture device 6 will be stored.

[0150] As mentioned earlier, the degree of positive importance can beused in a digital imaging system to rank images in a systematic andcontinuous manner as favorite, important or exciting images for aspecified user as described in commonly assigned U.S. patent applicationSer. No. 10/036,113, filed Dec. 26, 2001, entitled “Method for Creatingand Using Affective Information in a Digital Imaging System” byMatraszek et al.; and Ser. No. 10/036,123, filed Dec. 26, 2001, entitled“Method for Using Affective Information Recorded With Digital Images forProducing an Album Page” by Matraszek et al.

[0151] In another embodiment, different combinations of facialexpressions, eye characteristics and physiological reactions can be usedto create the personal affective tag to classify scenes in accordancewith a broader range of emotional categories, such as ‘joy’, ‘fear’,‘anger’, etc. An example of such classification is shown in Table 1.TABLE 1 Emotion classification based on combinations of facialexpressions, eye characteristics and physiological reactions Signals EyePhysiological Emotion Facial expressions characteristics reactions Joysmile, crinkled Opened eyelids, accelerated heart rate, skin arounddilated pupils, large GSR eyes corners direct gaze Fear pale skin,Widely opened accelerated heart rate, trembling lips, eyelids, fastaccelerated breathing chattering teeth eye-blink rate, rate, tightenedmuscle fixed gaze, tension, sweaty palms dilated pupils, Anger loweredbrows, Narrowed eyelids, deep and rapid flared nostrils, fixed gaze,breathing, increased horizontal blood pressure wrinkles over nose bridgetense-mouth Surprise raised eyebrows, Opened eyelids, large GSR openedmouth, Fixed gaze wrinkled brow and forehead Disgust wrinkled nose,Narrowed eyelids, decreased breathing raised nostrils, Averted gaze rateretracted upper lip, visible tongue, lowered brows Sadness lowered lips,Narrowed eyelids, flaccid muscles, cheeks and jaw tearing eyes,decreased breathing down gaze rate

[0152] Images can be further classified using a range of values forthese categories, such as strongly happy, somewhat happy, neutral andsomewhat sad, and strongly sad, etc.

[0153] The determined affective information in terms of the emotionalcategory is then stored as personal affective tag, which includes theuser identifier as part of the image metadata. It can also be stored ina separate file on the computer together with the image identifier andthe user identifier.

[0154] Referring to FIG. 7, there is shown a flow diagram illustratinganother embodiment of the present invention wherein physiological andfacial analyses are used in combination to determine affectiveinformation. In this embodiment, affective information about a user'sresponse to an image is based on the combination of the four affectivesignals described in relation to FIGS. 2-5, namely, the degree ofpreference, the degree of distinctiveness, the degree of interest, andthe degree of excitement, which are further combined to obtain anintegral measure of importance. This embodiment is suited for anembodiment such as a wearable image capture device 6 shown in FIG. 1c.

[0155] In this embodiment, method steps 510-528 correspond to methodsteps 410 through 428 in FIG. 6 with only two differences: the userselects the use of “combined distinct” signals or, alternatively, theimage capture device 6 is preprogrammed to use the “combined distinct”signals.

[0156] Image capture device 6 determines the degree of distinctiveness(DD) as described earlier in relation to steps 170-179 in FIG. 3 (steps530 and 533).

[0157] The image capture device 6 determines the degree of importance(or magnitude of emotional response). In this embodiment, the measure ofthe degree of importance is based on a sum of the four measures:

Importance=DP+DD+DI+DE

Importance=DPw _(DP) +DDw _(DD) +DIw _(DI) +w _(DE) DE

[0158] where the weights w_(DP), w_(DD), w_(DI), and w_(DE) aredetermined based on the standard deviation within each of the normalized(divided by the maximum value) signals previously obtained for theparticular user. In this case, the higher the standard deviation withinthe signal, the higher the weight of the contribution for the signalinto the measure of importance. Consequently, the lower the standarddeviation of a given signal, the lower the weight of the contributionfor the corresponding signal into the measure of importance. The reasonfor this dependency stems from the assumption that a standard deviationof a particular measure for a particular user reflects an individualdegree of differentiation between different scenes. This implies thatthe signal with the highest standard deviation has more differentiationpower, and therefore is more important to consider while determining anintegral measure of importance for a particular user.

[0159] For example, if different scenes evoke a large variations offacial expression and a low variation of skin conductance responses fora user A, than the weight given to the measure of degree of preference(DP) based on facial expression w_(DP) would be higher than the weightgiven to the measure of the degree of excitement (DE) based on skinconductance w_(DE). On the other hand, if different scenes evoke asmaller variation of facial expression and a large variation of skinconductance responses for a user B, than the relationships between theweights is reversed. Data about the maximum values and the standarddeviation of the corresponding signals can be obtained from the personaluser profile in. The personal user profile is then updated regardingthis information (step 543).

[0160] The obtained degree of importance is compared to a thresholdvalue (step 544). This threshold can be predetermined. This thresholdalso can be established by user 2 or established for user 2. If theobtained degree of importance is above the threshold value, the imagecapture device 6 creates a personal affective tag indicating the degreeof importance (step 548). In another embodiment the threshold valuecould be determined automatically from the personal user profile, forexample, on the basis of the prior cumulative probabilities of thedegree of importance distribution. One such threshold value then can bechosen as corresponding to the degree of importance with the priorprobability of 0.5.

[0161] The image capture device 6 stores the corresponding scene imageand the degree of importance in the personal affective tag as part ofthe image metadata (step 550). Alternatively, the degree of importancecan be stored in a separate file in association with the user identifierand the image identifier. In addition, the information about the user 2views a certain image can be also recorded as a separate entry intopersonal affective tag.

[0162] The corresponding image, the personal affective tag and otherimage metadata are sent to a personal database of digital images, asdescribed earlier in reference to step 152 of FIG. 2. (step 552).

[0163] If the obtained degree of importance is below the threshold valuethe corresponding segment of facial images, eye images and skinconductance signals are deleted (step 540). In another embodiment, ifthe obtained degree of importance is below the threshold value and user2 is still viewing the same scene or a captured image of the scene forexample on preview screen 22, image capture device 6 can optionallymeasure the next segment of facial images, eye images and skinconductance signals and repeat steps 532 through 544 as user 2 views thesame scene or the captured image of the scene.

[0164] If the threshold value is set to 0, all scene images andcorresponding affective information (degree of importance or, in anotherembodiment, raw facial images, eye images and galvanic skin responsesignals) recorded by the image capture device 6 will be stored.

[0165] In other embodiments, different combinations of these three orother affective signals (such as derived from voice, EEG, brain scan,eye movements and others) can be used to create the personal affectivetag to classify scenes in accordance with broader range of emotionalcategories. Further, non-affective information such as locationinformation, image analysis, calendaring and scheduling information,time and date information can be used to help better determine affectiveinformation such as a degree of importance for association with animage.

[0166] As mentioned earlier, the degree of importance can be used in adigital imaging system to rank images in a systematic and continuousmanner as favorite, important or exciting images for a specified user asdescribed in commonly assigned U.S. patent application Ser. No.10/036,113, filed Dec. 26, 2001, entitled “Method for Creating and UsingAffective Information in a Digital Imaging System” by Matraszek et al.;Ser. No. 10/036,123, filed Dec. 26, 2001, entitled “Method for UsingAffective Information Recorded With Digital Images for Producing anAlbum Page” by Matraszek et al. The degree of importance can also beused in an image capture device 6 to make decisions about imagecompression, resolution and storage. A computer program for creating andusing personal affective tags in an image capture device can be recordedon one or more storage medium, for example; magnetic storage media suchas magnetic disk (such as a floppy disk) or magnetic tape; opticalstorage media such as optical disk, optical tape, or machine readablebar code; solid-state electronic storage devices such as random accessmemory (RAM), or read-only memory (ROM); or any other physical device ormedia employed to store a computer program having instructions forpracticing a method according to the present invention.

[0167] The personal affective tag can also include informationdesignating a relative degree of importance. As described earlier, therelative degree of importance can be determined on the basis ofaffective information only. Alternatively, affective and non-affectiveinformation can be used in combination to determine the relative degreeof importance. Examples of non-affective information include date andtime information, location information such as would be available from aGlobal Positioning System or a similar type of electronic locator. Imageanalysis of the image itself can also be used as a source ofnon-affective information that can influence the relative degree ofimportance. For example, the presence of particular subject matter in ascene can be readily identified by existing image processing and imageunderstanding algorithms. One such algorithm is disclosed in commonlyassigned U.S. Pat. No. 6,282,317 BI entitled, filed Dec. 31, 1998 by Luoet al., the disclosure of which is incorporated herein by reference,describes a method for automatic determination of main subjects inphotographic images by identifying flesh, face, sky, grass, etc. as thesemantic saliency features together with the “structural” saliencyfeatures related to color, texture, brightness, etc., and then combiningthose features to generate belief maps.

[0168] Another image processing technique disclosed in commonly-assignedUS Pat. Pub. No. US2002/0076100A1 entitled “Image Processing Method forDetecting Human Figures in a Digital Image” filed on Dec. 14, 2000, byLuo et al., the disclosure of which is incorporated herein by reference,allows detecting human figures in a digital color image. The algorithmfirst performs a segmentation of the image into non-overlapping regionsof homogeneous color or texture, with subsequent detection of candidateregions of human skin color and candidate regions of human faces; andthen for each candidate face region, constructs a human figure bygrouping regions in the vicinity of the face region according to apre-defined graphical model of the human figure, giving priority tohuman skin color regions. The presence of people in a scene orparticular people, established using facial recognition algorithms suchas described in an article entitled “Face Recognition Using Kernel BasedFisher Discriminant Analysis”, published in the Proceedings of the FifthIEEE International Conference on Automatic Face and Gesture Recognition,pp. 0197-0201, 2002 by Liu, et al. may be used to increase the relativedegree of importance. It can also be used to selectively process theimage in order to enhance its quality, emphasize a main subject asdescribed in commonly assigned U.S. patent application Ser. No.09/642,533, entitled “Digital Image Processing System and Method forEmphasis a Main Subject of an Image”, filed on Aug. 18, 2000, by Luo etal. and to share this image with the people identified or to transmitthe image to an agency because of security concerns.

[0169] In the embodiments described above, the images and image capturesystem have been described as being digital images and digital imagecapture systems. Consistent with the principles of the invention, imagesof the scene can be captured in an analog electronic form or on anoptical medium such as a photographic film or plate. Where the image iscaptured in one of these forms, data representing affective informationcan be recorded in association with the image by recording affectiveinformation separately from the image with some identification codeindicating the image with which the information is to be associated.Alternatively, affective information can be encoded and recorded inassociation with the analog electronic image. Where a photographic filmor plate is used, the affective information can be recorded optically ormagnetically on the film or plate. The affective information can also berecorded on an electronic memory associated with the film or plate.

[0170] In accordance with the present invention, affective informationis described as being captured at the time of capture or during capture.As used herein, these terms encompass any time period wherein an imageis being composed or captured. Such time periods can also includeperiods immediately after the moment of capture, when, for example, aphotographer is verifying that a captured image meets her satisfaction.

[0171] The invention has been described in detail with particularreference to certain preferred embodiments thereof, but it will beunderstood that variations and modifications can be effected within thespirit and scope of the invention.

Parts List

[0172]2 user

[0173]4 scene

[0174]6 image capture device

[0175]8 capture module

[0176]10 user video camera

[0177]12 digital storage device

[0178]13 manual controls

[0179]14 central processing unit

[0180]15 sensors

[0181]16 galvanic skin response sensors

[0182]17 vascular sensor

[0183]18 communication module

[0184]19 vibration sensor

[0185]20 Internet service provider

[0186]22 preview screen

[0187]24 viewfinder

[0188]26 bridge

[0189]28 glasses frame

[0190]29 side piece

[0191]110 activate image capture device step

[0192]112 launch application step

[0193]114 enter user identification data step

[0194]116 determine emotional reaction step

[0195]118 select desirable signal step

[0196]120 capture scene image step

[0197]122 store scene image step

[0198]124 detect power on step

[0199]126 continue capture step

[0200]128 deactivate power step

[0201]130 capture facial image step

[0202]132 store facial image step

[0203]134 analyze facial image step

[0204]136 determine smile size step

[0205]138 determine degree of preference step

[0206]139 update personal user profile step

[0207]140 compare to threshold step

[0208]142 delete facial image step

[0209]144 create personal affective tag step

[0210]146 store image and affective tag step

[0211]148 send image and affective tag step

[0212]150 activate image capture device step

[0213]152 launch application step

[0214]154 enter user identification step

[0215]155 determine emotional reaction step

[0216]158 select desirable signal step

[0217]160 capture scene image step

[0218]162 store scene image step

[0219]164 detect power on step

[0220]166 continue capture step

[0221]168 deactivate power step

[0222]170 capture facial image step

[0223]172 store facial image step

[0224]174 analyze facial expression step

[0225]176 determine emotional category step

[0226]178 determine degree of distinctiveness step

[0227]179 update personal user profile step

[0228]180 compare degree of distinctiveness to threshold

[0229]182 delete image step

[0230]184 create personal affective tag step

[0231]186 store image step

[0232]188 send image and affective information

[0233]210 activate image capture device step

[0234]212 launch application step

[0235]214 enter personal information step

[0236]216 determine emotional reaction step

[0237]218 select eye gaze information step

[0238]220 capture scene image step

[0239]222 store scenic image step

[0240]224 detect power on step

[0241]226 continue capture step

[0242]228 deactivate power step

[0243]230 capture sample of eye gaze image step

[0244]232 store eye gaze coordinates step

[0245]234 determine fixation step

[0246]236 measure duration of fixation step

[0247]238 determine degree of interest in fixation step

[0248]239 update personal user profile step

[0249]240 compare to threshold step

[0250]242 delete image step

[0251]244 create personal affective tag step

[0252]246 store image and affective tag step

[0253]248 send image and affective tag step

[0254]310 activate image capture device step

[0255]312 launch application step

[0256]314 enter personal information step

[0257]316 determine emotional reaction step

[0258]318 select eye gaze information step

[0259]320 capture scene image step

[0260]322 store scene image step

[0261]324 detect power on step

[0262]326 continue capture step

[0263]328 deactivate power step

[0264]330 capture first segment of galvanic skin response step

[0265]332 store skin conductance step

[0266]334 filter galvanic skin response signal step

[0267]336 determine amplitude of galvanic skin response step

[0268]338 determine degree of excitement step

[0269]339 update personal user profile step

[0270]340 compare to threshold step

[0271]342 capture next segment step

[0272]344 store degree of excitement step

[0273]346 store image and affective tag step

[0274]348 send image and affective tag step

[0275]410 activate image capture device step

[0276]412 launch application step

[0277]414 enter personal information step

[0278]416 determine emotional reaction step

[0279]418 select desirable signal step

[0280]420 capture scene image step

[0281]422 store scene image step

[0282]424 detect power on step

[0283]426 continue capture step

[0284]428 end

[0285]430 capture facial images step

[0286]432 determine degree of preference step

[0287]434 capture eye segment step

[0288]436 determine degree of interest step

[0289]438 capture segment of skin conductance signals step

[0290]440 determine degree of excitement step

[0291]442 determine degree of importance step

[0292]443 query and update personal user profile step

[0293]444 compare importance to threshold value step

[0294]446 delete image step

[0295]448 create affective tag step

[0296]450 store images and affective tag step

[0297]452 send data step

[0298]510 activate image capture device step

[0299]512 launch application step

[0300]514 enter personal information step

[0301]516 determine emotional reaction step

[0302]518 select desirable signal

[0303]520 capture scene image step

[0304]522 store scene image step

[0305]524 end

[0306]526 continue capture step

[0307]528 deactivate power step

[0308]530 capture facial image step

[0309]532 determine degree of preference step

[0310]533 determine degree of distinctiveness step

[0311]534 capture eye segment step

[0312]536 determine degree if interest step

[0313]538 capture segment of skin conductance signals step

[0314]540 determine degree of excitement step

[0315]542 determine degree of importance step

[0316]543 query and update personal user profile step

[0317]544 compare importance to threshold value step

[0318]546 delete facial eye and galvanic skin step

[0319]548 create affective tag step

[0320]550 store images and affective tag step

[0321]552 send image step

What is claimed is:
 1. An imaging method comprising the steps of:capturing an image of a scene; collecting affective information atcapture; and associating the affective information with the scene image.2. The method of claim 1 further comprising the steps of: collectinguser identification data; and associating the affective information andthe user identification data with the scene image.
 3. The method ofclaim 1, further comprising the steps of: collecting user identificationinformation; identifying a user based on the user identification data;and associating the affective information and the scene image with theuser.
 4. The method of claim 1 wherein the step of associating theaffective information with the image comprises storing the scene image,the affective information and user identification data in a commondigital file.
 5. The method of claim 1, wherein the step of associatingthe affective information with the image comprises storing the affectiveinformation within the scene image.
 6. The method of claim 2, whereinthe affective information and user identifier are stored in associationwith the scene image.
 7. The method of claim 2 further comprising thestep of using the collected affective information and useridentification data to build a personal user profile.
 8. The method ofclaim 7 further comprising the step of: identifying a user based on theuser identification data.
 9. The method of claim 1 wherein the step ofcollecting affective information comprises monitoring the physiology ofthe user.
 10. The method of claim 1 wherein the collected affectiveinformation is used build a personal user profile.
 11. The method ofclaim 9, wherein the step of collecting affective information comprisesthe steps of interpreting the collected physiological information todetermine the relative degree of importance of the scene image.
 12. Themethod of claim 1, wherein the step of collecting affective informationat capture comprises collecting an image of a user from which affectiveinformation can be obtained.
 13. The method of claim 12, wherein anelectronic image of at least a part of a face of the user is capturedand affective information is derived therefrom.
 14. The method of claim1, wherein the step of collecting affective information at capturecomprises determining a relative degree of importance.
 15. The method ofclaim 14 wherein the step of associating the affective information withthe image comprises storing the relative degree of importance inassociation with the scene image.
 16. The method of claim 11 wherein therelative degree of importance of the image is established at least inpart based upon the personal user profile.
 17. The method of claim 1wherein the step of automatically collecting affective informationincludes monitoring the eye gaze of the user.
 18. The method of claim 1,further comprising the step of obtaining non-affective information atcapture.
 19. The method of claim 18, further comprising the step ofinterpreting the affective information and non-affective information todetermine the relative degree of importance of the scene image.
 20. Themethod of claim 19, wherein the non-affective information includes thecaptured image and the relative degree of importance is at least in partdetermined by analysis of the scene image.
 21. The method of claim 1wherein the step of collecting affective information comprisescollecting manually entered affective information.
 22. The method ofclaim 1 wherein the step of collecting affective information at capturecomprises detecting composition of an image, and collecting affectiveinformation during composition.
 23. The method of claim 1 wherein thestep of collecting affective information at capture comprises detectingverification of a scene image and collecting affective informationduring verification.
 24. An imaging method comprising the steps of:capturing an image of a scene; collecting affective signals at capture;determining a relative degree importance of the scene image based atleast in part upon the collected affective signals; and associating therelative degree of importance with the scene image.
 25. The method ofclaim 24, wherein the affective signals comprise physiologicalcharacteristics.
 26. The method of claim 24, wherein the affectivesignals comprise facial characteristics.
 27. The method of claim 24wherein the affective signals comprise physiological characteristics andfacial characteristics.
 28. A photography method, comprising the stepsof: capturing an image of a scene; obtaining an image of a photographerat capture; determining affective information based at least in part oninterpretation of the image of the photographer; and associating theaffective information with the scene image.
 29. The method of claim 28,wherein non-affective information is collected at capture and the stepof determining affective information further comprises determiningaffective information based at least in part upon the non-affectiveinformation.
 30. The method of claim 28, wherein information regardingthe physiology of the photographer is collected at capture and the stepof determining affective information further comprises determiningaffective information based at least in part upon the physiology of thephotographer.
 31. The method of claim 28, wherein information regardingthe physiology of the photographer is collected at capture and the stepof determining affective information further comprises determiningaffective information based at least in part upon the physiology of thephotographer.
 32. The method of claim 28, wherein the image of thephotographer is captured in the non-visible portion of the spectrum. 33.An imaging method comprising the steps of: capturing a stream of images;collecting a stream of affective information during image capture; andassociating the stream of affective information with the stream ofimages.
 34. The method of claim 33 wherein the stream of affectiveinformation includes user identification data.
 35. The method of claim33, wherein the stream of affective information comprises is examined todetermine when relevant changes in affective information occur and thestep of associating the affective information with the correspondingstream of images comprises associating data representing relevantchanges in the affective information with the stream of images at apoints in the stream of images that correspond to the occurrence of therelevant changes.
 36. A method for determining affective informationcomprising the steps of: obtaining affective signals including facialcharacteristics and physiological characteristics of a person; analyzingthe facial characteristics; analyzing the physiological characteristics;and, determining an emotional state based upon the analysis of thefacial and physiological characteristics of the person.
 37. The methodof claim 36 wherein the step of obtaining affective informationcomprises obtaining affective information at capture of an image. 38.The method of claim 37, further comprising the step of storing theaffective information with the captured image.
 39. The method of claim36 wherein the step of collecting affective information includes manualentering user's reaction.
 40. The method of claim 36 further comprisingthe steps of capturing an image of a scene being observed by the personat the time that the affective signals are obtained, wherein the step ofdetermining an emotional state is further based upon analysis of thescene image.
 41. An c imaging system comprising: an image capture systemadapted to capture an image selected by a user, a memory which storesthe image, and a set of sensors adapted to capture affective signalsfrom the user at capture, and a processor adapted to associate theaffective information with the captured image.
 42. The imaging system ofclaim 41 wherein the set of sensors is further adapted to capture useridentification data.
 43. The imaging system of claim 40, wherein theprocessor is adapted to determine a user identity based upon analysis ofthe user identification data.
 44. The imaging system of claim 43 whereinthe processor further comprises a transmitter transmitting a signalcontaining at least one captured image in association with affectiveinformation and the user identification data.
 45. The imaging system ofclaim 41 wherein the processor is further adapted to process affectiveinformation captured by the set of sensors to provide further affectiveinformation.
 46. The imaging system of claim 41 wherein the camerafurther comprises at least one source of non-affective information. 47.The imaging system of claim 46, wherein the processor is further adaptednon-affective information from the at least one source and determines adegree of relative importance based upon the affective information andthe non-affective information.
 48. The imaging system of claim 41further comprising a second image capture system adapted to capture animage of at least a part of the user.
 49. The imaging system of claim 41wherein the second image capture system captures an image of the user ina non-visible portion of the spectrum.
 50. The imaging system of claim48 wherein the processor is further adapted to analyze the image of atleast a part of the user to determine the identity of the user.
 51. Theimaging system of claim 48 wherein the processor is further adapted toanalyze the image of at least a part of the user to determine affectiveinformation therefrom.
 52. The imaging system of claim 41 wherein theprocessor is further adapted to analyze the image of the scene todetermine non-affective information therefrom and to determine a degreeof relative importance based upon the affective information and thenon-affective information.
 53. The imaging system of claim 43 whereinthe processor is further adapted to create and update a personal userprofile that comprises affective information.
 54. The imaging system ofclaim 53 wherein a personal user profile further comprises non-affectiveinformation.
 55. The imaging system of claim 41 wherein at least onesensor is adapted to collect affective information from signals carriedby the nervous system of a person.
 56. The imaging system of claim 41wherein said camera is packaged in a wearable form.